If you’re not good at taking care of pokemon, scared of pokemon, or just don’t like pokemon you’re usually looked down upon (See: Tori Lund from Deoxys movie).
If you are not good at battling (usually early in your journey) it is very possible that you will encounter violent wild pokemon and you could die. (See: Dawn, Serena, first day of their journey)
Even if you are good at training Pokemon, somewhere out there, some god damn ursaring is going to have its panties in a knot and will try to kill you (See: literally like every episode with a wild ursaring).
If you don’t want to work with pokemon, there are somewhat limiting job options available for you, and you may feel a bit out of the loop from everyone else (See: that doctor from the first series)
AND
If you are poor, and you don’t have pokemon growing up, and you associate with the wrong people, BECAUSE your jobs are limited, it’s very very possible that you end up working for a criminal organization (where you will probably steal, manipulate, or poach pokemon). (See: Jessie from Team Rocket, possibly other Team Rocket members)
Speaking of which, the baddies in pokemon are downright terrifying sometimes, and because it’s a kid’s show, I’m sure they’re capable of more that we just aren’t seeing. We know they’re not reluctant to perform MURDER (See: Hunter J, Team Rocket)
Considering the ‘journey’ aspect is what most people probably glorify from the series, journeying itself is harder when you’re not able-bodied. It’s hard if you are sick, weak or incapable of battling. (see: Wally from RSE) Surely you can get the help from pokemon, but from what I’ve seen, it’s pretty challenging.
Um, according to the novelizations, a lot of pokemon trainers actually are unsuccessful and become jobless bums because they have no education and they wasted their youth playing pokemon so yeah.
Have you read some of the Pokedex entries?? (there’s a post somewhere but UM…an episode with litwick showed them trying to send them to HELL)
YEAH, I’m sure there’s more, but basically…I think that every world should be looked at more holistically and by ignoring the downsides, it makes the pokemon world a bit one-dimensional and uninteresting. Not saying that people shouldn’t prefer it to our own world, but it’s kinda like people going to a foreign country and expecting it to be a utopia of sorts when really, they have very universal problems.