Mdthbs

I’m thinking that there will be one small continent, about half the size of Australia or a bit larger. The single, massive ocean is inhabited by an aquatic humanoid species (yes, think mer-people) that speak to one another in a way that is similar to the sounds that whales make. They do not communicate with the alien species that have settled on the planet’s lone continent via space travel.

The ‘land-loving’ aliens have learned that the mer-people (I haven’t come up with a name for them yet) don’t mind them fishing from shore or heavily fortified boats (I might end up changing this to hover crafts of some sort), but that they become hostile when the aliens attempt to swim in the ocean (this isn’t the only reason that the aliens avoid swimming: there are several species of carnivorous, gluttonous leviathans that inhabit the ocean). The mer-people live in harmony with the leviathans, but the alien species are attacked by them on sight (still thinking on why the leviathans leave the mer-people alone).

My question is: Where did this continent come from or how did it form on a planet that’s almost entirely encompassed by one massive ocean and how large a human or humanoid species population could inhabit such a small amount of land?

Also, if anyone sees anything unrealistic in anything that I’ve mentioned, please let me know.

It is worth noting that the difference between the deepest ocean trenches an highest mountains on Earth (less than 20 km) amounts to almost nothing compared to the planet's radius (6731 km). It is smoother than a ping-pong ball, relatively speaking. That we have roughly equal parts sea and land (70:30) is a happy accident that has probably aided life on Earth. According to astronomers, planets totally covered in tens of km of water are much more likely. Worlds with less oceans are also possible; Mars is though to have had an ocean on its north pole early in the solar system's history, comprising perhaps as much as a third of the surface. What I am saying with this is that a planet with a single, fairly small continent doesn't seem much more unlikely than what we see on our own planet or what we know has been the case on Mars.

The continent may be the result of tectonic activity that has raised a 'mountain' above the ocean surface. On our planet, all ground above 7 km is located in Asia, mostly in the Himalayas and Hindu Kush ranges (which are really parts of the same extended mountain range). If this is the case, expect your continent to be long and narrow, with steep sides.

It may also be that your planet has a patch with thin crust, resulting in high volcanic activity, which over billions of years has built a continent. The volcanic activity may have simmered down or ceased by the time of your story. Compare to Olympus Mons on Mars, which rises 22 km above the main Martian surface, covering an area roughly the size of France. This is just a single volcano; with half a dozen such, you could get a small continent. This continent will be flatter than one formed by tectonic forces, and probably more fertile, since that is usually the case with volcanic soil.

Finally (though less likely), your continent may be the result of a large comet impact, leaving a huge complex crater, probably a multi-ringed basin. This will result in several circular mountain ranges and possibly a central peak, with seas in between. Valhalla Basin on the Jovian moon Callisto has as radius of almost 2,000 km.

The continent can be the sole remnant of a large impact with a less dense celestial body.

Being less dense, the impactor, or what remained on it on the surface of the planet after the impact, did not sank completely into the body, and remained afloat within the boundaries given by gravity to its maximum elevation.

Incidentally, this is one of the proposed explanation of how the continental crust on our own planet formed.

Where did this continent come from or how did it form on a planet that’s almost entirely encompassed by one massive ocean

Continental drift was the theory that the Earth's continents have moved over geologic time relative to each other, thus appearing to have "drifted" across the ocean bed. But before all that happened, all the landmass was one continent - Pangaea. And before the Pangea, there may have been landmasses that came together to form it.

It is fair to assume that your merpeople are living in one such world, where the land actually drifts to form larger masses.

how large a human or humanoid species population could inhabit such a small amount of land

Since your planet is earth-like but with smaller "visible" land mass of 10%, whereas Earth has 30% of visible land mass, you can easily have a third of the population of earth at any point of time on the land. Note that this will vary with advancement of technology in society.

The mer-people live in harmony with the leviathans

You could look for reasons such as domestication, symbiotic relationship, complementary lifestyles, mer-people taste rotten :D, etc to build this up.

You don't have to explain where the continent came from - it's just how the planet formed. Earth had a super-continent - Pangea. The size of the continent is directly proportional to the amount of water the planet has. There is nothing strange in that, it's just natural occurrence. Given the wide variety of exoplanets we've detected, it completely expected that such planets exist.

One concern I would have is how exactly the merpeople breath underwater. Intelligent animals that live in the our oceans such as cetaceans rise to the surface to breathe, and are well adapted to holding their breath for a long time. But actual undersea animals such as fish just don't get enough oxygen from their gill structures to support the energy hungry brains that provide a platform for intelligence.

So perhaps the merpeople have to surface regularly, which of course would mean they need to live near the surface? Perhaps there are naturally occurring air filled caverns replenished by geological processes in which they mostly live? Or perhaps these processes cause massive bubbling throughout the whole ocean and the merpeople's bodies are adapted to capture these bubbles? This could lead to a fun conflict narrative where the land dwellers are mining the ocean and causing a reduction in this bubbling though some sort of fracking process. Or perhaps there is a special plant material that grows all over the ocean surface that eventually sinks, and when they eat it it provides a huge amount of oxygen. Perhaps merpeople, to survive, need to have a backpack full of this food at all time. (Again there is a great conflict narrative there.)

As to my last point, perhaps there is some substance in this plant food that when eaten makes them extremely toxic to the leviathans, and so natural selection has caused the leviathans to not eat or attack merpeople.

As to where the land came from, I think the idea of an asteroid collision creating the land is excellent. And you could tie it together with the substances in the asteroid mixing with sea water to produce the bubble effect I described above. And the land people who need to mine that material for their own survival are put in conflict with the merpeople because their mining damages their air they breath.

It might be interesting too if the giant land mass was in fact composed of a great archipelago of islands so that using the land actually required the land people to use the sea, with, again, the generation of conflict.

Just a few thoughts. Sounds like an interesting world.

1. There was once more land.

Your merpeople were originally land dwellers, which is why they are humanoid. Your planet originally had considerably more land, and water tied up in ice caps. With the gradual warming of the planet (greenhouse gas? sun heating up?) there was melting of the icecaps. The humanoids adapted to wetlands, then shallows, then ultimately became aquatic. There is a theory that some human characterists are a result of partial adaptation to wetland / shallows - the aquatic ape hypothesis. Take that further.

2. 
   Levitathians are adult merpeople
   
   Your merpeople are juveniles in a prolonged childhood. When they go through puberty they grow and become ravenous - the leviathans. These eat too much to live around the juveniles, who would be outcompeted for food, so they go off and live by themselves (like bull elephants or bull whales). They become hungry and huge, and they fight each other ferociously, but they do not become stupid. They do not eat their own kind.

This is something which would be revealed halfway through the book when your protagonist realizes 1: the leviathans are intelligent and later 2: they are also merpeople.