I don't see any reason not to do so. The real question is what type of farm do you want to have? Livestock? Poultry? Fruits? Vegetables? Fish? Nuts? Mushrooms? (the legal kind) Spices?
Unless you already have access to a lot of acreage, infrastructure and equipment it may be a bit hard to start farming grains (e.g. wheat, corn, soybeans, rice, etc.). From what I've seen in looking at how to start/grow my own farm that's one area of farming where there are some minimum sizes involved just due to the harvesting process since the smallest combines intended for production usage only get so small. There are smaller ones made for research farms/universities, but I'd be questioning just how easily/cheaply they can be supported and how efficient they are at harvesting.
Come to think of it my parents were in their late 30's/ early to mid 40's when they started raising chickens, ring-necked pheasants, and wild turkeys as a "hobby farm" (some animals were sold, but many were used to offset the grocery bills associated with having three sons). Likewise the neighbor they had for several years was older than that (think he may have been in his 50's?) when he started raising red deer (which were purely sold for commercial butchering & sale - even though he viewed/treated his primary herd rather like pets).
So from the stats I've seen, while most of the food grown in the US is grown on large farms; most farms in the US aren't large farms (i.e. the farm income isn't the sole income). Which means the portrayal of farmers being nothing but farmers isn't the reality across most of the US. While all sizes and types of farms play a role (and are needed),in some ways running a "small" farm can actually be advantageous in that it can be a bit more forgiving (though not always) and permit a bit more experimentation/learning while providing an additional source of income.
Parts of my mother's family do farm as their sole/primary occupation, and from what I've seen of that (and from conversations with coworkers who have similiar backgrounds) relying solely on farming for income is neither easy nor low-stress - even on a relatively large, multi-generation farm with a diversity of products. At the other extreme in size a great example of farming can be seen on Youtube in "Urban Farmer" Curtis Stone who built up a farming enterprise using his backyard and yard space rented/leased from others in the area to grow various quick growing greens.
So I guess my advice would be to look at what you do have available, what can be done where you are (e.g. urban/suburban areas may have ordinances against raising livestock within their borders),and then decide where you want to start. Even if you don't end up with a farm that's measured in square miles of area, as long as you enjoy it and it offsets the costs of doing it (or in some cases even if it doesn't but you really want to do it) then why should age play a part in making the decision?
I've known more than a few military officers who've retired after long military careers (20-30 years) who then decided to start farming just to have something more to do. ....and that neighbor with the red deer? He was doing it because they too had crossed the point where their children were adults, and it was something he wanted to do to occupy his time. I know it wasn't their sole source of income, but I'm pretty sure he was still making money doing it. So there's nothing wrong with starting small and growing it (or not) to get the size and type of farm you want.....
Just my $0.02 and there a lot of very knowledgeable individuals here that can provide a wide variety of perspectives....
