Around here probably the biggest threat would be racoons and black (rat) snakes which can grow quite large. I have seen about a 6’ long one around here. You’re right about enclosing the surrounding vegetation inside the aviary itself, making sure it is safe, you can just build the aviary around the greenery! Here is a closer picture of an aviary that we enclosed 3 small poplar trees inside of. Of course much of the trees have been chewed up but they’re still growing!
We had a similar issue with our aviary. The way we protected them was to have two layers of wire on both sides of each panel so that nothing could get them. Hope you understood my description, we don’t have these anymore otherwise I would upload a picture so you can see what I mean.