I think it is really dependant on your financial situation, as you have a young family, and it is nice to have that regular income and it can be a slow process building up your clients.
I, like yourself, was working for an Accountancy practise for many years and I know that this provides you with amazing experience of all types of businesses and Bookkeeping. I have seen on this forum newly qualified people who are taking the plunge to go self-employed with out any real experience, so I think you are in a great position from that point of view.
My family is grown up and I was in a position to have some savings behind me in case my earnings were not sufficient.
I went self employed 18 months ago and have been really pleased as I am now working less hours for more money and have a greater flexibilty.
I have obtained some of my client from job sites on the internet where small companies are looking for 1 day a week and I have 3 of those which I go to each week. (This guarentees a level of 3 days income every week which will sustain me and any other small jobs can be slotted in as and when and are a bonus). I have 4 monthly payrolls and 2 other monthly bookkeeping jobs and now I am just looking for another small client to be complete.
Smaller one man bands who only want self assesment once a year will not make enough money for you to replace a salary unless you get hundreds.
I have contacted Accountants but they are not that forthcoming as most offer bookkeeping services themselves and they do not want to pay the hourly rate that I charge.
So I would say that if your financial situation will stand it then you should have confidence in yourself and go for it (especially as you already have a couple of clients!)