Rockstar is known for creating games with some of the highest production values ever. While both the GTA and Red Dead Redemption series differ in quite a few meaningful ways, both series clearly have an absolutely humongous budget, and special care is taken in the look and feel of the final product.
Rockstar has dabbled in a lot of different genres, usually mirroring that of films. One genre that Rockstar hasn't done anything with thus far is sci-fi. But, with the way in which Rockstar incorporates film genre ideas into its games/narratives, it could make something really cool out of the sci-fi genre.
Rockstar Borrows A Genre's Best Ideas
Just to name a few genres, Rockstar has used the Western for Red Dead Redemption, both the Crime/Parody for Grand Theft Auto, and the Noire/Detective genres for L.A. Noire. For each of these, Rockstar has taken the usual formula that these genres provide, and elaborated it into what it would take to make a game out of it. After all, a movie is generally going to be between 90-180 minutes; a game is going to be much, much longer than that. Figuring out which aspects of the genre to flesh out, and where to expand upon it entirely, is crucial in making sure the games are a success instead of long-winded.
For a sci-fi game, Rockstar would have a wealth of inspiration to draw from. Looking at Rockstar's typical sensibilities, it would probably steer clear of the Star Wars and Flash Gordon's of the genre. More than likely, it would decide to go a more serious sci-fi route, more along the lines of something like Star Trek. A story of that format would be good inspiration anyway, simply because a TV show tells a story over a longer period of time than a movie, much like a game does.
A Rockstar Sci-Fi Game Would Break Out of the Mold
While there are quite a few sci-fi games on the market, there aren't really any sci-fi games that have the feel that a Rockstar game tends to. Many sci-fi games out there also fit into a sub-genre; Dead Space and Alien: Isolation, for example, both fit into sci-fi and horror. Eve Online and Elite: Dangerous both fit into sci-fi simulator and MMO.
Rockstar campaigns are typically single-player, open-world, narrative experiences with a ton of cutscenes and a long runtime. There's typically quite a lot of player freedom in the open-world, with a much more restricted amount of freedom during the missions. One of the only game series that seems like it's similar to what a Rockstar sci-fi game could look like is Mass Effect. Though Mass Effect does heavily rely on player choice, while Rockstar games don't generally have this mechanic outside of a few instances.
Rockstar also generally lets players play both on foot and in a vehicle; in a sci-fi game, that vehicle could be a spaceship. There aren't a lot of sci-fi games out there that allow for this, as many games prefer to stick entirely to on-foot gameplay or in-ship gameplay. Star Citizen (what's finished of it currently) and the upcoming Elite Dangerous: Odyssey update are notable exceptions to this.
Lately, Rockstar has also allowed for players to choose between a first-person and third-person perspective, an option that is pretty rare in a lot of games. This lets Rockstar corner both markets, as certain players find it a deal-breaker to not be allowed to play in one or the other. Especially in regards to space combat, this would be an option that a lot of players would appreciate.
Sci-fi With A Realistic Take
Rockstar generally puts a bit of thought into creating gameplay systems focused on realism; this is more the case in Red Dead Redemption 2 than any other Rockstar game, but examples can be found throughout its history. As just one example, in RDR2, the players' diet affects Arthur Morgan's physique. His physique then influences the gameplay, albeit slightly.
Taking this inclination towards realism into sci-fi games could bring some really interesting and inventive gameplay mechanics. Aboard a spaceship, for instance, there could be a pressure system. If the hull is breached or a door is opened, it could spell bad things for anybody inside. It could also create some really cool mechanics planetside; different planets could have vastly different properties, and those properties could realistically influence gameplay in a huge way, forcing players to adapt how they're playing to the planet that they find themselves on.
Rockstar's Online Component
One thing that GTA 5 and RDR2 both have in common is an outrageously successful online component. Both Red Dead Online and Grand Theft Auto Online function in the same map and engine as the single-player experience, but have entirely new stories, missions, and gameplay features than the single-player version. These have added hundreds of hours of content onto these two games without needing to spend time crafting whole expansions onto the campaigns, allowing the developers to focus more on gameplay than narrative.
Having this same kind of online component in a Rockstar sci-fi game could prove really fun. It would have a little bit of competition, depending on what exactly this game would be, but with the Rockstar name and general quality, it would probably prove successful. Rockstar's online games generally let the player choose what kind of character they would like to be, giving quite a few options at launch and expanding on them with regular updates from launch onwards. This kind of gameplay system really allows players to role-play in the impeccably detailed worlds that Rockstar creates; which is a lot of the reason these two online games have the staying power that they have. Players feel a certain ownership of their character because they've worked towards actually accomplishing something.