It's not possible to identify which Peter Hogg this is, even in England, from the information in the question.
So to track him back to another country you do need as many clues as you can gather, rather than (as too many do!) just picking the first person who could be a possible match.
Finding his marriage (likely pre-1837?) could help. If you have found the birthplace of the earliest child, that would a place to start looking for the marriage. If you do find it, the names of the witnesses could contain a relation, perhaps one that came down from Scotland for the event.
If any of the censuses contain any relations of his, that would also help. But I assume if you had already found a mother-in-law, you would have mentioned it.
If you have his death in England, that would be another indication of his age, to match against the censuses. There may also be probate or wills which mention relatives.
Follow each of his children forward to future censuses, to see if they ever stay with relations in Scotland, or have a Scottish "visitor" staying with them.
Lacking any of this, another approach is to identify all the likely Peter Hogg's in Scotland, and try to follow them forward (this does assume he was indeed born in Scotland). The Peter Hoggs that get (and remain) married to somebody else, and/or stay in Scotland, you can eliminate.
ScotlandsPeople shows only five Peter Hoggs born in Scotland from 1795-1810, so it's not an impossible task. In other words, research fully each of those five Peter Hoggs, finding their future lives and descendents. If you can find what happens to 4 of them but not the final one, you have a strong candidate.