Abstract: The rapid development of communication technology has significantly improved information transmission and increased capacity. In the context of the Internet of Things (IoT), where massive visual data transmission faces challenges of real-time processing and security threats. To address this problem, this paper proposes a novel encryption algorithm based on a 3D S-box combined with Fractal-Sort-Matrix and Fibonacci Q-Matrix (3DSFF). To address the shortcomings of traditional S-box encryption, this paper integrates the 3D S-box with the FSM, thus enhancing the uncertainty of permutation and transformation. Additionally, to overcome the limitation of using a single value inciting Fibonacci Q-Matrix (FQM) applications, this paper combines FQM with chaotic sequences to strengthen its resistance against exhaustive attacks. Through comprehensive experimental tests on the algorithm’s outcomes, it demonstrates notable improvements over previous algorithms, achieving an average information entropy of 7.9993 and a correlation coefficient close to 0.01 after encryption. These tests indicate that the scheme can withstand common attacks, making it a sufficiently secure solution for the confidentiality of private images. Moreover, these advances provide a secure and efficient visual data protection framework for IoT applications involving anti-theft surveillance, data acquisition, and communication transmission.