The Stewart name is well-documented and most Stewart families can be traced back to a William Stuart II (b. 1780, Fayette, North Carolina; d. 12 September 1860, Sparta, Tennessee) and Rachel Wisdom (b. 1785, Glouchester, North Carolina; d. 1850, White County, Tennessee), who in turn can be traced a couple generations back to England. Similarly, this particular branch of the Stewart line does have a relation to William II and Rachel, which most family trees online prioritize as a patrilinial connection. In reality, based on evidence and my own conjecture to fill in the gaps, it might just be a much more tenuous relationship.
What we can rely on as fact is this:
On 14 December 1845, John William Stewart (b. 1821, Tennessee) and Sarah Stewart (b. 1824, Tennessee) married in White County, Tennessee. They would have one son in 1848, Rufus Thomas Stewart, but would pass early in his life; Sarah in March 1850 in White County and John sometime between 1850 and 1860, after which Rufus lived with his Uncle John Byron Stewart (who has a much-historied family tree) from 1860 through 1880. Rufus' life from that point on is well-documented for this Stewart line.
As stated before, most family trees attribute John William Stewart as William II and Rachel's son (and John Byron's brother). While that would give a much cleaner and easier lineage, providing several already researched generations back to the family's English origins, I cannot find myself in agreement (much to my own dismay, let it be known!). Instead, I believe the relation with William II and Rachel is actually Sarah. I base this on three main points.
The first is sourced from John and Sarah's various marriage documents, in all of which she is listed as "Sarah Stewart." This is an odd detail compared to other marriage documents of the time and place, where the wives-to-be are otherwise graced with their maiden names. This implies her maiden name was already Stewart.
By the 1850 United States Federal Census, Sarah had passed, and John and Rufus are sharing home with a Anna Stewart (age 42) and Barsheba (age 17) and Abraham (age 14) Stewart. There's an obvious relation, though hard to determine to which of the parents it is to. Perhaps Anna was a parent of either John or Sarah, though she would have been around 13 for the former's birth and 16 for the latter. This isn't impossible, but I'm dubious. Might she be an older sister? The 13 year gap between her and John would be much more reasonable if that were the case. Moving in with a brother's family during hard times seems much more likely than moving in with a dead sister's family, too. This isn't as firm as the first point, but supplies an amount of secondary supporting evidence.
Third is the obvious relationship between Rufus and John Byron, as he was taken in and listed as a nephew in the 1860 Federal Census, though is the relation through John or Sarah? Let's go backwards in census data.
In the 1850 Federal Census, William II and his family members are listed as:
William Stewart, age 70 (patriarch)
Rachel Stewart, age 65 (matriarch)
John Byron Stewart, age 31 (son)
Lavina Stewart, age 21 (wife to John Byron)
Stephen Stewart, age 22 (son)
Rachel Stewart, age 20 (daughter)
Malvina Stewart, age 19 (daughter)
as well as two young boys (William and Rachel's grandchildren).
Thus, for the 1840 census, it would be expected to see 6 people: William and Rachel, and their 4 children:
William Stewart, age 60
Rachel Stewart, age 55
John Byron Stewart, age 21
Stephen Stewart, age 12
Rachel Stewart, age 10
Malvina Stewart, age 9
While the 1840 Federal Census does not include a list of names, it does include genders and age ranges:
Free White Persons - Males - 10 thru 14: 1 (Stephen)
Free White Persons - Males - 50 thru 59: 1 (William)
Free White Persons - Females - 5 thru 9: 1 (Malvina)
Free White Persons - Females - 10 thru 14: 1 (Rachel)
Free White Persons - Females - 15 thru 19: 1
Free White Persons - Females - 50 thru 59: 1 (Rachel)
John Byron is missing, though being 21, it is not surprising he was not included in his father's household. There is also a additional older teen girl, a spot that would fit Sarah, who would be 16 years old at the time, while John William (age ~19) has no slot to fit into.
Going further back to 1830, a total of 5 is expected:
William Stewart, age 50
Rachel Stewart, age 45
John Byron Stewart, age 11
Stephen Stewart, age 2
Rachel Stewart, age 1
The 1830 Census has:
Free White Persons - Males - Under 5: 1 (Stephen)
Free White Persons - Males - 10 thru 14: 1 (John Byron)
Free White Persons - Males - 15 thru 19: 1
Free White Persons - Males - 50 thru 59: 1 (William)
Free White Persons - Females - Under 5: 1 (Rachel)
Free White Persons - Females - 5 thru 9: 2
Free White Persons - Females - 10 thru 14: 1
Free White Persons - Females - 40 thru 49: 1 (Rachel)
This has several additional spaces, but Sarah (age 6) would fit in perfectly, while John William (age 8-9) has no spot available.
In the end, my personal conclusions are that it is Sarah who is the sibling of John Byron; Anna being John's elder sister and not his mother. The dates and ages make sense with this arrangement, though it does make determining the patrilineal line any further back much more difficult, if it's even possible. Still, it is for these reasons that I feel solid in using this inferred information as fact for this family tree.