Joseph's brothers had already faced him once, but they didn't know he was their brother. They didn't know that Joseph actually had good plans for them, so the thought of returning made the brothers very nervous.