The modern Olympic Games were inaugurated in 1896. In 1900, the games of the Second Olympiad were held in Paris. At that time, France ruled an extensive colonial empire encompassing Indochina and much of Africa, without the consent of native populations. The games were neither boycotted nor protested.
In 1904 the Olympics were held in St. Louis, U.S.A. At the time, the U.S. was wrapping up its military campaign against a native insurrection in the Philippines, which this country had taken from Spain after the 1898 war. Historians estimate that American troops killed hundreds of thousands of Filipinos during the insurrection. The games were not boycotted.
In 1908 the Games were held in London. This was during the time when the sun supposedly never set on the British Empire. Great Britain's was the largest of the European colonial empires. King Edward VII was also Emperor of India, without the consent of the Indian people. The games were not boycotted.
In 1920 the Games were held in Antwerp, Belgium. Even little Belgian had an Empire, ruling the "Belgian Congo" which had once been the personal property of the Belgian King and was proverbial for murderous exploitation. The games were not boycotted.
In 1924 the Summer Olympics were once again held in France, and once more not boycotted.
In 1928, the Olympics came to Amsterdam. At that time, the Netherlands ruled Indonesia and a number of smaller colonies, without the native peoples' consent. The games were not boycotted.
In 1932 the Summer games were held in Los Angeles. The U.S. still ruled the Philippines, and would do so, not counting the Japanese occupation during World War II, until 1946. The games were not boycotted.
In 1936, many people did object to the idea of Adolf Hitler hosting the Olympics in Berlin, but the games were not boycotted.
In 1948, the first Summer Olympics after World War II were held in London. While Great Britain had recently divested itself of India, it still ruled an increasingly restive empire extending from Malaya to Hong Kong to Kenya, and the games still were not boycotted.
It's nice to suppose that we are simply more enlightened than our insensitive ancestors, and that there's no double standard involved when people object to China hosting the Olympics, or when they objected to Moscow hosting them. The issue, however, isn't whether anyone has a double standard, because a great gulf of years separates the latest of these colonial-era Olympiads from the boycott era. The point of this history lesson is to have you ask: what's changed? I'd suggest that ideology has crept in here as it has in so many places, and that ideologues would find any excuse to deny the legitimacy allegedly conferred upon rival ideologies by hosting the Olympic Games. You'll probably see the same thing if a Muslim country gets to host the Games, presuming that such a country wants them. The Olympic ideal, of course, is a kind of ideology unto itself, and that, too, may be why Americans like to talk about boycotts so often.