"System" is a term that is flung around willy-nilly in the Star Wars hexalogy1. It is never quite clear from its usage in the movies what is meant by the term, and the hosts and guests have often speculated back and forth as to its the meaning. However, it is an actual term used in astronomy2,3. It refers to any body in space around which natural satellites orbit. A 'Star system' is a specific type of astronomical system where two or more stars orbit each other or a central point, whereas a 'planetary system' is one in which non-stellar objects orbit a star. The Solar system is an example of a planetary system, and consists of our sun Sol, and all the planets, planetoids, asteroids, and comets that orbit it. The Earth and its moon (Luna) can also be thought of as a system, the Earth-Moon 'satellite system'. This was discussed in ROTJ Minute 105.
However, this doesn't help us determine the naming conventions for systems in the Star Wars universe. The hosts have long debated whether the system is named for the Star or the main planet of interest. Names are inconsistently applied, much like the names of neighborhoods in Los Angeles, as mentioned in ESB 82.
From what we've been able to parse by pouring through Wookieepedia, it would appear that Systems are indeed named after the star (although the primary planet of interest may also be named after the star, which is confusing), and planets are numbered from innermost to outermost (as opposed to the order in which they were discovered).