As of April 2021, the global gaming value exceeded $300 Billion. While a substantial chunk of it is due to the rapid adoption in the recent pandemic and the availability of a plethora of mobile games, gaming is no more a child’s play. A sustainable, long-term monetization strategy can’t just be an afterthought but an essential building block for gaming, media, and entertainment businesses.
It’s 2022, and we are entering a new dimension with bleeding-edge innovations across industry verticals. The line between gaming, the media and entertainment is getting thinner by the day. Seamlessly integrated experiences may have sounded fancy a couple of years back, but now it’s a bare essential.
We’re about to enter the next generation of seamless connected-tech experiences, thanks to Metaverse. It’s time for content-based enterprises, from small indie studios to large AAA studios and OTT platforms, to reconsider their monetization strategy.
The Current State of Monetization
Monetization in games, media, and entertainment has evolved over the decades.
Games were sold as finished products in cartridges and then on physical discs. They used to be one-and-done products for which the customer got the most for their spend.
With the shift towards digital game stores like Steam, Epic, GOG, and others, the monetization landscape saw a transformational shift.
For once, studios and publishers saved a lot of money since they didn’t have to print, package, and transport the physical game copies. They could also rely on fixing game bugs and patching them via Over-The-Air updates post-release, and then there’s Downloadable Content.
All these had a budget constraint and limited the possibilities for smaller players.
However, the paradigm shift in monetization came with mobile games. They brought in a shock and awe effect by leveraging the freemium models and using in-app purchases, in-game ads, and paid games to bypass the ads and paywalls compared to free games.
In media and entertainment, the advent of streaming platforms saw a similar dynamic, and the typical commercials saw a shift. Traditional media houses were left behind with a significant gap as OTTs ruled the roost.
But now, we are about to witness another shift that will affect the current monetization strategies of the gaming, media, and entertainment industries.
The New Era of The Internet is Changing Everything
We have too many digital game stores and streaming platforms, and the users are starting to feel the pinch. Subscription fatigue is starting to set in, making monetization challenging for businesses.
We now have blockchains in the mix, which can change the whole economy of monetization for both creators and businesses. Like it or not, the metaverse might very well become a crucial player in the world economy.
Physical and Virtual Lives will Bridge for a Unified Experience
XR (Extended Reality), an amalgamation of Virtual Reality, Augmented Reality, and mixed reality technologies, will pave the way for our future entertainment content requirements.
So, what does this mean for businesses? Big Tech giants rule with an iron fist, and content moderation is a grey area in the current climate.
“With the next generation of the internet, we are looking at decentralization and a leap of technology,” said Christopher Lafayette, Founder and CEO at Gatherverse, when he recently spoke at a virtual summit.
Advertising has already changed since its inception, and today it’s focused on content creators and consumers.
With content creators and influencers, advertising has taken center stage and helps ads find takers among the form of consumers with their large subscriber base.
Non-Fungible Tokens (NFTs) are now trending and are set to be a digital form of payment, letting users buy and trade digital assets. With users creating communities for such “digital marketplace,” the playing field for monetization is in dire need of an update and can’t rely on traditional practices.
What is the Future of Monetization?
The future of monetization with web 3.0 may be questionable, but the majority of web3.0’s focus is on decentralization. User is at the core; and will be the driving force, be it content, ads, or monetization. Adaption is the need of the hour for businesses.
Sure, the traditional payment methods will remain. But businesses have to acknowledge the fact that blockchain will be thrown into the mix and change the dynamic of the digital economy. Rafael Brown, CEO & Co-Founder at Symbol Zero, who was a speaker in a recent tech symposium, said, “PC and Mobile gaming have established a monetization economy. As technology changes with time, we need to revisit our assumptions. The need of the hour for blockchain technology is to create sustainable monetization.”
The tech summit brought together more than 20 world leaders from Gaming, Media & Entertainment to participate and unravel the direction we as humans powered by tech are headed with web 3.0. With discussions on monetization, metaverse, subscription fatigue, OTT platforms, and many more interesting topics, the virtual event was a hit around the globe.
Affine combines the hyper-convergence of AI, data engineering & cloud with deep industry knowledge in manufacturing, gaming, CPG, and technology. Affine demonstrates thought leadership in all relevant knowledge vectors by investing in research through its highly acknowledged centers of excellence and strong academic relationships with reputable institutions.
Reach out to us to put a robust and sustainable monetization strategy in place for your business!
Gaming went from being a niche hobbyist culture to becoming a reckoning mainstream phenomenon. Today we have close to 3 billion gamers around the globe, which answers the simple question of the need for analytics in gaming for businesses.
The plethora of data! That’s a rich repository of insights to be leveraged.
The gaming industry was worth $178 billion in 2021, projected to exceed $268 billion by 2050. Gaming has shifted its weight on cloud technology, for its backend services over the years. Mobile gaming turned out to be a successful phenomenon, onboarding even casual players and turning it into a lucrative segment.
With the pandemic, gaming skyrocketed in demand. Mobile games saw such a spike that even the streaming giant Netflix has decided to dip its toes in mobile gaming. Who can blame them?
The mobile gaming market will cross $272 billion by 2030, according to Global Data!
Smiling young man playing game on smartphone isolated over yellow background, looks happy and concentrated, looking smiling at his device’s screen, spending spare time indoor, guy holding mobile phone
With such high stakes, there are a couple of behind the scene factors attributing to the smooth functioning and success of the video game industry.
Let us dig deeper and understand how cloud technology will accelerate AI (Artificial Intelligence) analytics adoption in the video game industry.
What is Gaming Analytics?
On average, an online gaming company generates close to 50 Terabytes of data in just 24 hours. Managing such a significant player base is no easy task, and we’ve already seen how the cloud functions as the bloodline for backend gaming infrastructure and managed services.
Such a mammoth repository of gameplay data may contain vital elements that game studios can leverage using gaming analytics.
Gaming analytics may mean many things to many organizations. The use cases are far too many with every service, but game studios need to understand their respective problem statements.
In broader terms, gaming analytics leverages the user data from games for business decisions, marketing activities, and product improvement.
Use cases of analytics for gaming include improving the game design, monetization for increased revenue, effective marketing strategy, and many more.
The 4 Steps of Gaming Analytics
There are stages to analytics called analytics maturity models that describe them based on their criticality. Let us classify these into four stages:
Stage 1: Descriptive Analytics
Descriptive analytics is a solution when a gaming organization is trying to answer the question, “What happened?”
Descriptive analytics through the depiction of available data provides an understanding of the present situation, giving a realistic view of current events and potential opportunities.
The effectiveness of power-ups, health packs, and save points in certain game levels can be determined using descriptive analytics.
Stage 2: Diagnostic Analytics
Diagnostic analytics is the solution to the question, – “Why did this happen?”
It is used in determining the relationship between two variable elements by analyzing historical game data. One of the most vital outputs of diagnostic analytics is to find an organization’s effectiveness, “How are the results compared to the efforts?”
Why does one of the players get such a lower score than the others in one of the challenges? Diagnostic analytics provides the answers, even for end gamers in games like Hitman, where there are parameters for completing particular achievements.
Stage 3: Predictive Analytics
We are now entering the meatier part of game analytics. Predictive analytics falls under the advanced stage territory of analytics. Unlike descriptive and diagnostic analytics, predictive analytics plugs in and tracks gamer behavior in real-time.
By creating forecasts, predictive analytics can measure and foresee the consequences of various actions.
There are patterns and trends in the gameplay data that can be used to understand the elements that make the game great and areas where improvement is necessary.
On the marketing side, predictive analytics can help gaming businesses identify target users for user acquisition; all these factors help optimize their marketing budget.
With the marvel of predictive analytics, gaming companies can foresee their business scenarios as active participants, making sustainable and profitable business decisions while providing excellent gameplay to players, resulting in increased play-time and in-game spending.
Stage 4: Prescriptive Analytics
Prescriptive analytics is the final boss of the analytics game. The stages of analytics till now can provide a gaming business with tons of information, painting a picture that tells a story.
Prescriptive analytics leverages these stages and helps businesses take action, turning the data into results with fruition.
While machine learning and artificial intelligence are just thrown around everywhere on the internet, prescriptive analytics uses machine learning algorithms to make optimum decision recommendations.
There is no need to wonder about the right course of action. Prescriptive analytics uses artificial intelligence and machine learning algorithms to run millions of simulations with precision and recommends the best outcome, leveraging the prowess of Big Data.
Prescriptive analytics is so robust that it can dive deep and shed clear light on data patterns and trends. Not only is this useful in games, but it can also help understand the pulse of consumers and help companies improve customer engagement.
Game Analytics is a Driving Force for Game Studios
A plethora of challenges plagues the present-day video game industry. Studios have to deal with a million variables in player preferences and design mission levels that engage the vast userbase across the globe.
The gameplay experience is one area where game analytics will be the game-changer. Studios always face the uphill task of designing the perfect difficulty levels for their games. Developers have to develop their game levels to maintain the right balance between challenge and progression.
Game UI is also an area that studios cannot overlook. It impacts the gameplay experience and affects the time spent by the player.
Game analytics may seem overkill at first glance, but we live in a data-fueled generation with an ever-increasing dependency on the internet. We saw what the pandemic did to many industries. It also helped us understand the importance of being ready for the unknown.
Gaming is one industry that had the fortune of seeing an upward trend during the pandemic, making an even more compelling argument for implementing game analytics.
With the increased number of gamers, spending on games has also increased, and game studios must leverage the potential. To do that, every decision must be weighed not only in terms of business optimization but also in providing value to the players. Understanding the player sentiment is the key to cracking the monetization strategy, non-intrusive and engaging ads, while tapping into potential player spending using fair practices, resulting in increased ROI.
Game analytics is not a fancy service. From lndie to large AAA, every type of studio can leverage game analytics to have a foolproof system in place, guiding them with well-informed business decisions resulting from precision with no room for human error.
Indie games were once for a niche crowd that enjoyed the retro-styled game design and mission progression. Low-key passionate developers who made the quality-of-life mods for popular games, or some brave heart developer who would try to cook up a game with an archaic code and a sub-par computer, made indie games back then. Even if someone managed to make an indie game with the minimum resources, there would be no buzz around the game since no major studio entertained such games.
Today, the Indie game scene is thriving and mainstream, with a sizeable audience. A lot has evolved to an extent that the term “Indie games” has been redefined.
Games like Hollow Knight, Terraria, and Among Us weren’t just chartbusters; they made the Indie space a mainstream one with a solid audience and substantial foothold in the market. What further echoes the success of Indie games is the fact that their development requires a significantly lower budget compared to mainstream AAA, RPG, and Sports games. Their pricing is competitive, aimed at garnering more takers and relying on mass sales volumes. Raking high revenue with a low budget is a guaranteed blueprint for maximizing ROI. But with a limited budget, Indie developers have a set of daunting challenges developing and releasing games in the long run.
The Rise of Indie Games
Many factors have contributed to the rise of Indie games.
The unsung hero here is the digital download platforms powered by cloud backend that make game downloads easier and more economical while helping indie games foray into the average gamer’s library.
Unlike large studios, Indie developers fell short of the budget to churn out physical copies in copious amounts, restricting the reach and exposure of their games no matter how good.
With Steam, GOG, EPIC, and other digital storefronts backed by cloud infrastructure and data centers worldwide for local downloads, it is just a matter of uploading their games to the cloud and letting gamers across the world download them.
Digital platforms became mainstream, and indie developers didn’t have to worry about manufacturing expenses owing to making physical game copies eating into their profits.
Indie Marketing – A Challenge That Cloud Can Address
We see a regular churning out of tons of indie games. The segment is one of the most competitive. For successful indie titles like Celeste and Minecraft, hundreds of games go unnoticed.
Multiple factors are at play here.
Marketing is one key aspect where Indie studios lack the resources to reach a global audience. Like with large studios, volume-based marketing may be impossible for small studios, but effective data-based marketing is not only possible but imperative.
Efficient targeting is the lifeline of successful marketing. Bombarding social media platforms with ads without a proper marketing plan will not result in user acquisition or brand engagement. You must understand your target base before going all out on digital marketing.
Marketing and user acquisition in this day and age is an omnichannel affair. Video games have a global audience, and studios cannot afford to overlook this factor.
Improving the reach of games requires a well-crafted marketing plan that covers all grounds.
Analytics-based marketing is the core to ascertaining the audience base for game studios. The keyword here is data-collecting data at every avenue, and online touchpoints are vital to understanding customer behavior and patterns.
But it is also the first step toward the logical marvel of data-powered analytics
Data Collection, Analytics, and ROI (Return on Investment)
With the advent of digital marketing, the term “marketing” is thrown around like confetti, and small, less experienced businesses like indie game studios find it challenging as much is lost in translation.
Indie studios must focus on collecting vital data like game metrics to understand their players’ behavior and data from social media platforms that can help them tune their marketing campaigns for maximum efficiency.
Combining the accrued data with cloud analytics helps studios with ROI-based marketing, giving them a real-time view of what’s going on with their marketing activities instead of a “what went wrong?” meeting.
This way, real-time tweaks to the digital marketing efforts are made possible, and studios can focus on targeting prospective leads on particular social platforms rather than unnecessarily advertising on every platform and user.
Since indie studios have a limited marketing budget, effective ROI-based marketing is the ideal balance to help achieve brand recognition and user acquisition.
Game Analytics – A Must-have Solution for Indie Game Studios
We saw the advantages of cloud-based analytics solutions for marketing. Now, let us look at game analytics that can help indie studios tweak their games and release updates and patches that result in increased player engagement and game lifetime overall.
Game metrics can help studios offer users a dynamic gameplay experience.
Player fatigue, monotonous gameplay, and challenging levels play spoilsport in the long run and affect the gameplay duration of players. DLC and additional content can help here, but it involves resources like budget, development, and workforce.
With game analytics, studios can read real-time game metrics, analyze them, and improve gameplay so that players will put more hours into the game. All this comes without worrying about the additional development cost and human resources for DLC.
There are plug-n-play game analytics solutions that overcome the infrastructural challenges and initial setup costs. Even the maintenance is done on the service provider’s end so that studios can focus on improving the player gameplay experience, the true intent of these solutions rather than logistics.
Cloud Solutions for Indie Game Studios are Not Only Imminent, but Imperative
Even the Silicon Valley giants have acknowledged that the cloud is paving the way for gaming and have introduced multiple cloud-based gaming services.
In 2020, Google introduced a managed service program called Game Servers. Unlike Stadia, Game Servers isn’t a game streaming service but a backend cloud server infrastructure that helps game developers build and scale backend servers for their game titles. So, unlike the common misconception that the cloud in gaming only means streaming, the cloud also plays a vital role in acting as a backend infrastructure for all types of games.
With a considerable stake in gaming and its recent acquisition of Activision Blizzard, Microsoft has revealed its new product, ID@Azure, which lets indie game developers develop their games from scratch for the cloud platform.
Large studios have already made this transition and are reaping its benefits, not only in terms of cost but also in the elimination of managing the backend infrastructure. Why spend time and resources on it, when many service providers like AWS (Amazon Web Services), Microsoft Azure, Google and others can take care of it while developers adapt to the subscription cost model and focus only on making quality games?
A German indie mobile game studio, Coldfire games, eliminated its backend management efforts due to adopting an efficient cloud infrastructure. It is the only time before other indie developers follow suit and adopt cloud infrastructure as a backend to their games.
Then we have the cloud gaming phenomenon, which is set to gain significant traction over the next few years, with players like Netflix getting into gaming and giants like Google and Nvidia having services in place. Just like streaming became mainstream, cloud gaming will set a benchmark in mainstream gaming, and developers need to adapt to this, providing games at a subscription model.
From marketing to game analytics and backend infrastructure, cloud solutions are highly beneficial to game studios, particularly Indie developers who don’t necessarily come equipped with the resources, team, and technical know-how for game marketing and analytics. But it can be challenging to choose the right ally for your game analytics requirements. Affine is a pioneer player in the AI analytics arena, working with large and indie gaming organizations to synthesize game analytics into efficient and effective business outcomes
The passion for developing exceptional games is all an indie studio needs, for cloud-based solutions are plenty with simple implementation techniques and measurable ROI-based results that will assist them in every step.
Gaming is one of the fastest evolving industries, with considerable technological advancements. We’ve come from retro arcade games to LAN parties and playing portable smart gadgets on the go.
In terms of graphics, we saw the evolution from pixel art to 2d, and then 3d models. While PC has always been able to do justice to the visually exquisite-looking titles, current-generation consoles have changed the dynamics of the graphics race. Ultra-realistic graphics with reflective surfaces and interactive environments give users an immersive gaming experience. All this before even dipping toes into Virtual Reality, which is yet to hit mainstream status.
The pandemic witnessed a sudden spike in the interest in gaming. People flocked to play games, which served as an interactive pastime compared to streaming shows and movies. The aftermath? The gaming industry raked in revenue of $156 billion as of September 2021, and the global video game market is forecasted to cross $200 billion as of 2023!
Cloud is the Backbone of the ‘Always Online’ Culture
The lack of proper infrastructure in the previous generation limited gaming options to local play, offline games, and LAN parties at best.
Massively Multiplayer Online games were a pipe dream back then. But with the turn of the millennium, things changed.
Small data centers became multiple server farms with global CDN (Content Delivery Network) for maximum scalability.
Multiplayer games have gained significant traction over the years and are now a norm. Today we have a plethora of multiplayer online titles, with single-player games providing multiplayer options for players who want to explore beyond the original storyline.
Titles like Rainbow Six Siege made bank for Ubisoft with lifetime sales of over 1.1 billion as of 2021, with a player base of 70 million as of 2021.
The cloud infrastructure in place for a feat like this speaks for itself, which would have been impossible a decade back, simply owing to the lack of tech infrastructure.
Single-player games significantly depend on the cloud infrastructure thanks to the paradigm shift from DVDs to digital game stores like Steam, Epic Games, and GOG. Pre-loading games, cloud saves, day one updates, and DLCs are standard practices in the gaming industry now. The days of waiting in long lines for days before the next GTA release is a thumbnail in history. With millions of players downloading at a time across the globe, sustainable cloud infrastructure is at the heart of gaming infrastructure.
Gaming on Demand will be a Reckoning Force in the Future of Gaming
Gaming on-demand or gaming as a service is growing by the day and will shape the path of gaming as it did with content consumption via streaming. Steam link, Nvidia GameStream PS4 RemotePlay and many such services offer gaming to the end-user on reasonably fast internet. Gaming hardware bundled with free games and discounts still comes with the invisible baggage of limitations owing to short cycle yearly tech upgrades in the gaming industry, nullifying the economy factor in gaming.
Developing games for multiple platforms is also an arduous, time-consuming task for game developers, which results in inconsistent gameplay experiences for the player. We’ve seen releases like Watchdogs, CyberPunk2077 looking graphically inferior in their release versions compared to the announcement versions. While many factors are at play here, the challenge to develop games for a previous generation platform alongside the next generation of consoles with high-end hardware causes compatibility issues in the game build and adds to the development cycle. Gamers also have to wait a long time since the release dates keep getting pushed to accommodate fixes to the build.
The high upfront cost for purchase and scarcity of vital hardware components like storage, RAM, and Graphic Cards create an opportunist and hobbyist culture in gaming. Thankfully, the advent of Gaming on-demand will address the issue. While it may take some time for global adoption, the end result not only provides every gamer the opportunity to high-speed gaming, game makers get access to the end-users without the hindrance or hardware barrier.
Cloud Gaming will be the Netflix of Games
While Netflix is testing waters and entering the gaming domain with mobile games, cloud gaming is an underlying phenomenon that will transform the industry and cement its position as the next chapter in gaming.
Stadia may have fallen short of wowing the gaming community, but there are perils to being an early adopter. For one, the number of mobile gamers in recent years has seen an alarming spike thanks to PvP games like PUBG and Fortnite captivating the masses and converting non-gamers into serious, habitual gamers.
This opens the doors for cross-platform play, which is currently a pipe dream with rare working examples, but all this will change with the cloud technology at the center stage.
With cloud gaming at the helm, the industry envisions a platform-agnostic gaming ecosystem powered by high-speed internet, primarily relying on robust cloud infrastructure for inclusive, sustainable and affordable gaming.
Currently, there is a global hardware draught from the pandemic’s disruption of manufacturing & supply chain, in addition to scalpers grabbing available stocks. This is a testament to how the dependency on hardware for gaming has reached a saturation point.
By eliminating the storage and graphical requirements that are a roadblock for many aspiring gamers, cloud gaming (gaming on demand) brings the biggest USP to the table for gamers, developers and studios. Gamers don’t have to break their bank going on a hardware shopping spree for next-gen graphics or order terabytes of high-performance Solid-State Drives for games that cross the 100GB mark.
Developers don’t have to fret over the game’s performance fidelity across multiple platforms or downgrade the graphics so that players can achieve decent frame rates across different types of devices. There’ll also be a significant reduction in the development period, which could help release games as per the dates advertised!
Studios can increase their target audience from hardcore gamers to even new players since the hardware barrier is no longer an issue. A feasible subscription gaming model will onboard a significant number of new gamers, and studios are looking at a rise in their user base with an imminent increase in ROI and a sustainable business model.
Cloud gaming emphasizes safety which is vital in online gaming. Player information leaking on the internet is nothing new, and even the top-of-the-crop studios like Bethesda have had this misfortune with Fallout 76. Cloud-based gaming models have bulletproof online security, which reduces the chances of player databases getting breached by external attacks and information leaks across the web.
Conclusion
Spending for modern hardware every 2-3 years at inflated astronomical prices in the current economy is not practical for players. Not to mention the pile-up of hardware junk which is not sustainable for a green future. A subscription model, in the long run, is a better alternative.
For businesses, the cost advantage is obvious. Setting up new infrastructure that requires frequent upgrades is a messy and costly affair that entails enormous resource consumption. The big cloud players offer these services for a fraction of the cost with unlimited scalability options and an ‘only pay for what you use’ model.
Cloud is not all hype. We’ve seen its role in streaming, and the technology has shaped up the OTT platforms today, providing access to quality content within the press of a button to users across the globe. The question is how long until it becomes mainstream in the gaming space. Sure, hardware-based gaming will not go extinct. But a shift towards cloud gaming is imminent.
Cloud and gaming go hand in hand. If the present is anything to go by, the dependency on cloud is only going to go up with time.
“Cinema is a mirror by which we often see ourselves.” – Alejandro Gonzalez Inarritu
If 500 people saw a movie, there exist 500 different versions of the same idea conveyed by the movie. Often movies reflect culture, either what the society is or what it aspires to be.
From “Gone with the Wind” to “Titanic” to “Avengers: Endgame,” Movies have come a long way. The Technology, Scale, and Medium might have changed, but the essence has remained invariable to storytelling. Any idea or story that the mind can conceive can be given life in the form of Movies.
The next few minutes would be a humble attempt to share the What, Why, and How of the Movie Industry.
Sections:
The process behind movies
How did the movie industry operate?
How is it functioning now?
What will be the future for movies?
1. The process behind movies
We see the end output in theatres or in the comfort of our homes (on Over-The-Top (OTT) platforms like Netflix, Amazon Prime, HBO Max, etc.). But in reality, there is a long and ardent process behind movie-making.
As with all things, it starts with an idea, an idea that transcribes into a story, which then takes the form of a script detailing the filmmaker’s vision scene by scene.
They then pitch this script to multiple studios (Disney, Universal, Warner Bros., etc.). When a studio likes this script, they decide to make the film using its muscle for pre-production, production, and post-production.
The filmmaker and studio start with the pre-production activities such as hiring cast and crew, choosing filming locations to set constructions. Post which the movie goes into the production phase, where it gets filmed. In post-production, the movie gets sculpted with music, visual effects, video & sound editing, etc.
And while moviemakers often understand the art of movie-making, it becomes a product to be sold on completing production. Now they have to sell this story to the audience.
Marketing & distribution step in to ensure that this product is now available to its worldwide audience by promoting it in all possible mediums like billboards and social media. After this, it’s all about delivering an immersive, entertaining experience where your surroundings go dark and the mind lights up.
An Overview Of The Movie-Making Process
In recent times some creative flexibility has been observed in the above process. For example, studios who own certain characters or intellectual property like Marvel & DC characters hire the right talent to get their movies made. In such cases, big studios control significant aspects of the film, from content creation to distribution.
2. How did the movie industry operate?
For a considerable period, movies used to stay in theatres for a long time post their initial release before reaching other available forms of Home Entertainment (based on the technological choices/era). For example, Let’s take a movie that acted as the bridge between two distinct generations. Titanic transformed the industry from the old school blockbusters to the new school global hits (with technology, CGI, worldwide markets, etc.) And before the 2010s, blockbuster movies like Titanic used to run in theatres for several months. Titanic was the undisputed leader of Box Office for nearly four months, both in terms of the number of tickets sold and worldwide revenue generated.
Post its theatrical run of approximately four months, blockbuster titles used to be available in-Home Entertainment (HE) formats (such as DVD, VCD, etc.) These formats were available in various options based on the decade or era. Options such as rental or purchasable DVDs ruled the HE domain for a considerable amount of time. Until the emergence of the internet.
The Dawn of the internet brought in other sources of entertainment in competition to the traditional Movies, Sports, etc. These options gave the consumer alternate forms of entertainment (which resulted in shortened theatrical runs, approximately three months or less). They gave the studios also another platform to sell their content. Hence the Home Entertainment release windows were fast-tracked as a natural consequence to capitalize the most from the movie’s potential.
The following is an example of the pre-2020/pandemic norms in Hollywood.
December 25: Movie releases in Theatres (TH). Ex: Fast and Furious March 10 19: EST (Electronic Sell Through) release of the movie (Ex: Amazon, iTunes)
April 2: iVOD/cVOD (Internet/Cable Video on Demand) release of the movie (Ex: YouTube, Comcast, Amazon)
April 30: PST (Physical Sell Through) release of the movie (Ex: DVDs, Blu-ray discs)
After this, the movie becomes available on Linear TV networks (Ex: HBO)
An Overview Of Movie Releases Before And After Pandemic
3. How is it functioning now?
Amid all the uncertainty surrounding the COVID pandemic, the movie industry did come to a halt, as did many other integral aspects of people’s lives. Around March 2020, most theatres worldwide shut down to prevent the widespread pandemic. The forceful shutting of the movie industry immobilized crucial aspects of the filmmaking process, such as the filming & theatrical release of movies. Since it was not safe for people to gather in numbers, theatres closed, as did other forms of entertainment such as Concerts, Live Sports, etc. This change of unprecedented magnitude was the first since world wars, where major entertainment activities worldwide were shut down.
With every problem, there lies an opportunity, as with this change, innovation was the name of the game. Those businesses that innovated survived, and the rest were likely to perish. The founding stone for this innovation was laid a long time back. With the influx of the internet, OTT (Over-The-Top) & VOD (Video on Demand) platforms were rapidly growing. OTT Platforms like Netflix & Amazon Prime were significant players in the US and worldwide before the beginning of the pandemic itself.
Shutting down of theatres meant some movies slated for 2020 waited for release dates. In the movie industry, movies are often planned well in advance. Major studios are likely to have tentative release dates for the upcoming 2 to 3 years. Delaying movies of the current year not only does it cumulatively delay the subsequent year’s release dates, but it also decays the potential of the film (due to factors like heavy competition later, loss of audience interest, etc.)
Major studios & industry leaders lead the way with innovation. A new format (Premium Video on Demand) and a new release strategy were the most viable options to ensure the movie’s release, guaranteeing both financial and viewership success.
The New Format – PVOD (Premium Video on Demand) was essentially releasing the iVOD/cVOD rental formats at an earlier period by shortening the pre-pandemic normal of 12 weeks post-theatrical release window to an earlier release window.
There were two ways of doing this; the first one is a Day and Date release in PVOD, which meant the audience can watch a new movie (Ex: Trolls World Tour, Scoob!) on its first release date at the comfort of their homes via the PVOD/rental channels (Ex: Amazon/iTunes)
The second way for the PVOD format is by releasing the movie in PVOD 2 to 8 weeks post its release in theatres. This happened once people got used to the new normal during the pandemic. Theatres across the world opened partially with limited seating capacity (50%). This meant that a movie would release in theatres exclusively first (as it was previously). However, the traditional Home Entertainment window of 12 weeks bypassed to release PVOD at an early window of 2 to 8 weeks post Theatrical release. This was the key in catering to a cautious audience during the pandemic between 2020 to 2021. This enabled them to watch a newly released movie at the comfort of their homes within a couple of weeks of its initial release itself.
A similar strategy was also tried with EST, where an early EST release (Premium EST or PEST) is offered to people at an early release window. The key difference is that PEST and PVOD were sold at higher price points (25% higher than EST/iVOD/cVOD) due to their exclusivity and early access.
The other strategy was a path-breaking option that opened the movie industry to numerous viable release possibilities – a direct OTT release. A movie waiting for its release & does not want to use the PVOD route due to profitability issues, or other reasons can now release the film directly on OTT platforms like Netflix & Amazon Prime. These platforms, which were previously producing small to medium-scale series & movies, now have the chance to release potential blockbuster movies on their platform. Studios also get to reach Millions of customers across the globe at the same time by jumping certain cumbersome aspects posed by the conventional theatrical distribution networks (which includes profit-sharing mechanisms). In this route of OTT platform release, there are many advantages to all parties involved (Studios, OTT Platforms & Audiences) and the number of potential customers.
The studios either get a total remuneration paid for the movie upfront (e.g., Netflix made a $200 Million offer for Godzilla vs. Kong to its producers, Legendary Entertainment & Warner Bros.). Or get paid later based on the number of views gathered in a given period or a combination of both (depending upon the scale & genre of the movie). The OTT platforms will now have a wide array of the latest movies across all genres to attract & retain customers. The people will now get to watch new movies on their preferred OTT platforms at their convenience and get a great value for money spent (OTT 1-month subscription ~$10 for new movie + existing array of movies & series vs. ~ $10 Theatre Ticket Price for one movie)
Overview of Major OTT Platform in US
Given there are two new gateways (OTT & PVOD) to release films in addition to the existing conventional mediums such as Theatres, EST, iVOD, cVOD, PST. There are numerous beneficial ways a movie can be released to reach the maximum people & make the most profit for the filmmakers & studios.
Release Strategy Examples
In the above example, releasing a movie directly in OTT in parallel to theatrical release attracts more subscribers to the OTT platform and covers the traditional theatrical audiences.
In the second example, let’s take a direct to Home Entertainment approach, targeting audiences directly in PVOD & early OTT releases. Similar to the movies that were released during the pandemic, like Trolls World Tour & Scoob!
The third example shows a possibility where a movie can leverage all existing major platforms for a timely release.
Since there are hundreds of possibilities for any studio or filmmaker to release their movies, how would one know the best release strategy for a movie? Does one size fit all methods work? Or do we scale and change release strategies according to the Budget/Content of the movie? Are certain genre films more suited for large-scale theater experience than being better suited for Home Entertainment? Who decides this? That should be a straightforward answer. In most cases, the one who finances the film decides the release strategy. But how would they know what combination ensures the maximum success to recoup the amount invested and guarantee a profit for all involved?
In such an uncertain industry, where more movies fail than succeed (considering the bare minimum of breaking even), the pandemic-induced multiple release strategies compound the existing layers of complexity.
In an ocean of uncertainties, the ship with a compass is likely to reach the shore safely. The compass, in this case, is Analytics. Analytics, Insights & Strategy provide the direction to take the movie across to the shores safely and profitably.
Analytics, Insights & Strategy (AIS) helps deal with the complex nature of movies and provides a headstrong direction for decision making, be it from optimal marketing spend recommendations to profitable release strategies. There are thousands of films with numerous data points. When complex machine learning models leverage all this data, it yields eye-opening insights for the industry leaders to make smart decisions. Capitalizing on such forces eases the difficulties in creating an enjoyable & profitable movie.
4. What will be the future for movies?
The Entertainment industry evolves as society progresses forward. Movies & theatres have stood the test of time for decades. There will always be a need for a convincing story, and there will always be people to appreciate good stories. Although with what seems to be a pandemic-induced shift into the world of online entertainment & OTT’s. This change was inevitable and fast-tracked due to unexpected external factors.
What the future holds for this industry is exciting for both the filmmakers and the audiences. The audiences have the liberty to watch movies across their preferred mediums early on, rather than the conventional long drawn theatrical only way. The studios now have more ways to engage audiences with their content. In addition to the theatrical experience, they can reach more people faster while ensuring they run a profitable business.
We will soon start seeing more movies & studios using the OTT platforms for early releases and the conventional theatre first releases with downstream combinations of other Home Entertainment forms to bring the movie early to the audience on various platforms.
On an alternate note, in the future, we might be in a stage where Artificial Intelligence (AI) could be generating scripts or stories based on user inputs for specific genres. An AI tool could produce numerous scripts for filmmakers to choose from. It is exciting to think of its potential, for example, say in the hands of an ace director like Christopher Nolan with inputs given to the AI tool based on movies like Tenet or Inception.
Post-Pandemic, when life returns to normal, we are likely to see star-studded, big-budget movies directly being released on Netflix or HBO Max, skipping the conventional theatrical release. Many filmmakers have expressed concerns that the rise of OTT may even lead to the death of theatres.
That said, I do not think that the theatres would perish. Theatres were and will always be a social experience to celebrate larger-than-life movies. The number of instances where people go to theatres might reduce since new movies will be offered in the comfort of their homes.
With all this discussion surrounding making profitable movies, with the help of Analytics, Insights & Strategy, why don’t filmmakers and studios stop after making a couple of profitable movies?
The answer is clear, as stated by Walt Disney, one of the brightest minds of the 20th century, “We don’t make movies to make money, we make money to make more movies.”
Back in 2014, Ian Goodfellow and his colleagues presented the much famous GANs(Generative Adversarial Networks) and it aimed at generating true to life images which were nearly unidentifiable to be outcome of a network.
Researchers found many use-cases where GANs could entirely change the future of ML industry but there were some shortcomings which had to be addressed. ProGAN and its successors improve upon the lacking areas and provide us with mind blowing results.
This post starts at understanding GAN basics and their pros and cons, then we dive into architectural changes incorporated into ProGAN, StyleGAN and StyleGAN2 in detail. It is assumed that you are familiar with concepts of CNN and overall basics of Deep neural nets.
Let’s Start-
Quick Recap into GANs —
GANs are Generative model which aims to synthesize new data like training data such that it is becomes hard to recognize the real and fakes. The architecture comprises of two networks — Generator and Discriminator that compete against each other to generate new data instances.
Generator: This network takes some random numbers/vector as an input and generates an image output. This output is termed as “fake” image since we will be learning the real image data distribution and attempt to generate a similar looking image.
Architecture: The network comprises of several transposed convolution layers aimed at up-scaling and turning the vector 1-D input to image. In below image we see that a 100-d input latent vector gets transformed into (28x28x1) image by successive convolution operations.
Discriminator: This network accepts generator output + real image(from training set) and classifies them as real or fake. In the below image, we see the generator output is fed into discriminator and then classified accordingly by a classifier network.
Both the networks are in continuous feedback, where the generator learns to create better “fakes” and discriminator learns to accurately classify “fakes” as “fake”. We have some predefined metrics to check generator performance but generally the quality of fakes tells the true story.
Overall GAN architecture and its training summary-
? Note: In the rest of the article Generator and Discriminator networks will be referred as G network and D network.
Here is the step-by-step process to understand the working of GAN model:
Create a huge corpus(>30k) of training data having clean object centric images and no sort of waste data. Once data gets created, we perform some more intermediate data prep steps(as specified in official StyleGAN repository) and start the training.
The G network takes a random vector and generates images, most of which will look like absolutely nothing or will be worse at start.
D network takes 2 inputs (fakes by G from step 1 + real images from training data) and classifies them as “real” or “fake”. Initially classifier will easily detect the fakes but once the training commences, G network will learn to fool the classifier.
After calculation of loss function, D network weights are being updated to make the classifier stricter. This making predicting fakes easy for D network.
Thereafter the G network updates its parameters and aims to improve the quality of images to match the training distribution with each iterative feedback from D network.
Important: Both the networks train in isolation, if the D network parameters get updated, G remain untouched and vice-versa.
This iterative training of G & D network continues till G produces good quality images and fools the D confidently. Thus both networks reach a stage known as “Nash equilibrium”.
? Limitations of GAN:
Mode collapse — The point at which generator produces same set of fakes over a period is termed as mode collapse.
Low-Res generator output— GANs work best when operated within low-res image boundaries(less than 100×100 pixels output) since generator fails to produce images with finer details which may yield high-res images. Thus high-res images can be easily classified as “fake” and thus discriminator network overpowers the generator network.
High volume of training data — Generation of fine results from generator requires lot of training data to be used because less the data more distinguishable the features will be from output fake images.
Let us start with knowing the basics of ProGAN architecture in next section and what makes it stand out.
Vanilla GAN and most of earlier works in this field faced the problem of low-resolution result images(‘fakes’). The architecture could perfectly generate 64- or 128-pixels square images but higher pixel images were difficult to handle (images above 512×512) by these models.
ProGAN (Progressive Growing GAN) is an extension to the GAN that allows generation of large high-quality images, such as realistic faces with the size 1024×1024 pixels by efficient training of generator model.
1.1 Understanding the concept :
Progressive growing concept refers to changing the way generator and discriminator model architectures train and evolve.
Generator network starts with less Convolution layers to output low-res images(4×4) and then increments layers(to output high-res images 1024×1024) once the last smaller model converges. Similarly D network follows same approach, starts with smaller network taking the low-res images and outputs the probability. It then expands its network to intake the high-res images from generator and classify them as “real” or “fake”.
Both the networks expand simultaneously, if G outputs 4×4 pixel image then D network needs to have architecture that accepts these low-res image as shown below –
This incremental expansion of both G and D networks allows the models to effectively learn high level details first and later focus on understanding the fine features in high-res (1024×1024) pixel images. It also promotes model stability and lowering the probability of “mode collapse”.
We get an overview of how the ProGAN achieves generation of high-res images but for more detail into how the incremental transition in layers happens refer to the two best blogs —
ProGANs were the first iteration of GAN models that aimed at generating such high-res image output that gained much recognition. But the recent StyleGAN/StyleGAN2 has taken the level too high, so we will mostly focus on these two models in depth.
ProGAN expanded vanilla GANs capacity to generate high-res 1024-pixel square images but still lacked the control over the styling of the output images. Although its inherent progressive growing nature can be utilized to extract features from multiple scales in meaningful way and get drastically improved results but still lacked the fineness in output.
Facial features include high level features like face shape or body pose, finer features like wrinkles and color scheme of face and hair. All these features need to be learnt by model appropriately.
StyleGAN mainly improves upon the existing architecture of G network to achieve best results and keeps D network and loss functions untouched. Let us jump straight into the additional architectural changes –
Instead of directly injecting input random vector to G network, a standalone mapping network(f) is added that takes the same randomly sampled vector from the latent space(z) as input and generates a style vector(w). This new network comprises of 8 FC (fully connected)layers which outputs a 512-dimension latent vector similar in length to the input 512-d vector. Thus we have w = f(z) where both z,w are 512-d vectors. But a question remains.
What was the necessity to transform z w
“Feature entanglement” is the reason we need this transformation. In humans dataset, we see beard and short hair are associated with males which means these features are interlinked, but we need to remove that link (so we see guys have longer hair) for more diverse output and get control over what GANs can produce.
The necessity arises to disentangle features in the input random vector so as to allow a finer control on feature selection while generating fakes and the mapping network helps us achieve this mainly not following the training data distribution and reducing the correlation between features.
The G network in StyleGAN is renamed to “synthesis network” with the addition of the new mapping network to the architecture.
“You might be wondering? how this intermediate style vector adds into the G network layers”. AdaIN is the answer to that
2. AdaIN (Adaptive Instance Normalization):
To inject the styles into network layers, we apply a separately learned affine operation A to transform latent vector win each layer. This operation A generates a separate style y[ys, yb] (these both are scalars)from w whichis applied to each feature map when performing the AdaIN.
In the AdaIN operation, each feature map is normalized first and then scale(ys) + bias(yb) is applied to place the respective style information to feature maps.
Using normalization, we can inject style information into the G network in a much better way than just using an input latent vector.
The generator now has a sort of “description” of what kind of image it needs to construct (due to the mapping network), and it can also refer to this description whenever it wants (thanks to AdaIN).
3. Constant Input:
“Having a constant input vector ?”, you might be wondering why….?
Answer to this lies in AdaIN concept. Let us consider we are working on a vanilla GAN and off-course we require a different input random vector each time we want to generate a new fake with different styles. This means we are getting all different variations from input vector only once at start.
But StyleGAN has AdaIN & mapping network which allows to incorporate different styles/variations in input vector at every layer, then why we need a different input latent vector each time? Why can’t we work with constant input only?
G network no longer takes a point from the latent space as input but relies on a learned constant 4x4x512 value input to start the image synthesis process.
4. Adding Noise:
Need to have more fine-tuned output that looks more realistic? A small feature change can be added by the random noise being added to input vector which makes the fakes look truer.
A Gaussian noise (represented by B) is added to each of the activation maps before the AdaIN operations. A different sample of noise is generated for each block and is interpreted based on scaling factors of that layer.
5. Mixing regularization:
Using the intermediate vector at each level of synthesis network might cause network to learn correlation between different levels. This correlation needs to be removed and for this model randomly selects two input vectors (z1 and z2) and generates the intermediate vector (w1 and w2) for them. It then trains some of the levels with the first and switches (in a random split point) to the other to train the rest of the levels. This switch in random split points ensures that network do not learn correlation very much and produces different looking results.
Training configurations: Below are different configurations for training StyleGAN which we discussed above. By default Config-F is used while training.
2.1. Let us have a look on Training StyleGAN on Custom dataset:
Pre-requisites– TensorFlow 1.10 or newer with GPU support, Keras version <=2.3.1. Other requirements are nominal and can be checked on official repository.
Below are the steps to be followed –
1. StyleGAN has been officially trained on FFHQ, LSUN, CelebHQ datasets which nearly contain more than 60k images. So looking at the count, our custom data must have around 30k images to begin with.
2. Images must square shaped(128,256,512,1024) and the size must be chosen to depend upon GPU or compute available for training model. 3. We will be using official repository for training steps. So let us clone the repository and start with next steps.
4. Data prep — Upload the image data folder to clone repository folder. Now we need to convert the images to TFRecords since the training and evaluation scripts only operate on TFRecord data, not on actual images. By default, the scripts expect to find the datasets at datasets/<NAME>/<NAME>-<RESOLUTION>.tfrecords
5. But why multi-resolution data Answer lies in the progressive growing nature of G and D network which train model progressively with increasing resolution of images. Below is script for generating TF-records for custom dataset –
Source for the custom source code format. Carbon.sh
6. Configuring the train.py file: We need to configure the train file with our custom data TFRecord folder name present in datasets folder. Also there are some other main changes(shown in image below) related to kimgs and setting the GPUs available.
Train script from StyleGAN repo with additional parameter change comments
7. Start training — Run the train script withpython train.py. The model runs for several days depending on the training parameters given and images.
8. During the training, model saves intermediate fake results in the path results/<ID>-<DESCRIPTION>. Here we can find the .pkl model files which will be used for inference later. Below is a snap of my training progress.
Snapshot from self-trained model results
2.2. Inference using trained model: 1. Authors provide two scripts — pretrained_example.py and generate_figures.py to generate new fakes using our trained model. Upload your trained model to Google Drive and get corresponding model file link.
2. pretrained_example.py — Using this script we can generate fakes using different seed values. Changes required to file shown below –
3. generate_figures.py — This script generates all sample images as shown in StyleGAN paper. Change the model url in the file and if you have used different resolution training images make changes as shown below. Suppose you trained model on 512×512 images.
2.3. Important mention:
Stylegan-encoder repository allows to implement style-mixing using some real-world test images rather than using seeds. Use the jupyter-notebook to implement some style-mixing and playing with latent directions.
StyleGAN surpassed the expectation of many researchers by creating astonishing high-quality images but after analyzing the results there were some issues found. Let us dive into the pain points first and then have a look into the changes made in StyleGAN 2.
Issues with StyleGAN-
1. Blob(Droplet) like artifact: Resultant images had some unwanted noise which occurred in different locations. Upon research it was found that it occurs within synthesis network originating from 64×64 feature maps and finally propagating into output images.
This problem occurs due to the normalization layer (AdaIN). “When a feature map with a small spike-type distribution comes in, even if the original value is small, the value will be increased by normalization and will have a large influence”. Authors confirmed it by removing the normalization part and analyzing the results.
2. Phase artifacts: Progressive nature of GAN is the flag bearer for this issue. It seems that multiple outputs during the progressive nature causes high frequency feature maps to be generated in middle layers, compromising shift invariance.
StyleGAN2 like StyleGAN uses a normalization technique to infiltrate styles from W vector using learned transform A into the source imagebut now the droplet artifacts are being taken care of. They introduced Weight Demodulation for this purpose. Let us investigate changes made –
The first image(a) above shows the synthesis network from StyleGAN having 2 main inputs — Affine transformation (A) and input noise(B) applied to each layer. The next image(b) expands the AdaIN operation into respective normalization and modulation modules. Also each style(A) has been separated into different style blocks.
Let us discuss the changes in next iteration(image C)-
As seen in image above, we transform each style block by two operations-
1. Combine modulation and convolution operation(Mod) by directly scaling the convolution weights rather than first applying modulation followed by convolution.
Source
Here w — original weight, w’ — modulated weight, si — scaling value for feature map i.
2. Next is the demodulation step(Demod), here we scale the output feature map j by standard deviation of output activations (from above step) and send to convolution operation.
Here small ε is added to avoid numerical operation issues. Thus entire style block in now integrated into single convolution layer whose weights are updated as described above. These changes improve training time, generate more finer results, and mainly remove blob like artifacts.
2 Lazy Regularization:
StyleGANs cost function include computing both main loss function + regularization for every mini-batch. This computation has heavy memory usage and computation cost which could be reduced by only computing
regularization term once after 16 mini-batches. This strategy had no drastic changes on model efficiency and thus was being implemented in StyleGAN2.
3 Path Length Regularization:
It is atype of regularization that allows good conditioning in the mapping from latent codes to images. The idea is to encourage that a fixed-size step in the latent space W results in a non-zero, fixed-magnitude change in the image. For a great detailed explanation please refer —
Progressive nature of StyleGAN has attributed to Phase artifacts wherein output images have strong location preference for facial features. StyleGAN2 tries to imbibe the capabilities of progressive growing(training stability for high-res images) and implements a new network design based on skip connection/residual nature like ResNet.
This new network does not expand to increase image resolution and yet produces the same results. This network is like the MSG-GAN which also uses multiple skip connections. “Multi-Scale Gradients for Generative Adversarial Networks” by Animesh Karnewar and Oliver Wang showcases an interesting way to utilize multiple scale generation with a single end-to-end architecture.
Below is actual architecture of MSG-GAN with the residual connections between G and D networks.
Source
StyleGAN2 makes use of the different resolution features maps generated in the architecture and uses skip connections to connect low-res feature maps to final generated image. Bilinear Up/Down-sampling is used within the G and D networks.
PPL values improves drastically in all combinations with G network having skip connections.
Residual D network and G output skip network give best FID and PPL values and is being mostly used. This combination of network is the configuration E for StyleGAN2.
5 Large Networks:
With all above model configurations explained, we now see the influence of high-res layers on the resultant image. The Configuration E yields the best results for both metrics as seen in last section. The below image displays contribution of different resolutions layers in training towards final output images.
The vertical axis shows the contribution of different resolution layers and horizontal axis depicts training progress. The best Config-E for StyleGAN2 has major contribution from 512 resolution layers and less from 1024 layers. The 1024 res-layers are mostly adding some finer details.
In general training flow, low-res layers dominate the output initially but eventually it is the final high-res layers that govern the final output. In Config-E, 512 res-layers seem to have more contribution and thus it impacts the output too. To get more finer results from the training, we need to increase the capacity of 1024 res-layers such that they contribute more to the output.
Config-F is considered a larger network that increases the feature maps in high-res layers and thus impacts the quality of resultant images.
3.3. Let us have a look on Training StyleGAN 2 on Custom dataset:
Pre-requisites– TensorFlow 1.14 or 1.15 with GPU support, Keras version <=2.3.1. Other requirements are nominal and can be checked on official repository.
Below are the steps to be followed – 1. Clone the repository — Stylegan2. Read the instructions in the readme file for verifying the initial setup on GPU and Tensorflow version.
2. Prepare the dataset(use only square shaped image with power of 2) as we did in StyleGAN training and place it in cloned folder. Now let us generate the multi-resolution TFRecords for our images. Below is the command –
3. Running training script: We do not need to change our training file, instead we can specify our parameters in the command only.
4. Like StyleGAN, here too our results will be stored into ./results/../ directory where we can see our model files(.pkl) and the intermediate fakes. Using the network-final.pkl file we will try generating some fakes with some random seeds as input.
3.4. Inference with random seeds:
Upload your trained model to Drive and get the download link to it.
We will be using run_generator.py file for generating fakes and style-mixing results.
In the first command, we provide seeds from 6600–6625 which generates 25 fake samples from our model corresponding to each seed value. Thus we can change this range to get desired number of fakes.
Similarly for style-mix, there are row-seeds and col-seeds input which generate images for which we need to have style-mixing. Change the seeds and we will get different images each time.
3.5 Results:
This sums up the StyleGAN2 discussion covering important architectural changes and training procedure. Below are the results generated and it gets rid of all issues faced by StyleGAN model.
Check out this great StyleGAN2 explanation by Henry AI Labs YouTube channel –
Thanks for going through the article. With my first article, I have attempted to cover an important topic in Vision area. If any error in the details is found, please feel free to highlight in comments.
References –
Custom code snippets: Create beautiful code snippets in different programming languages at https://carbon.now.sh/. All above snippets were created from the same website.
In the gaming industry, the studios are responsible for the design and development of any game. A game designer will be responsible for providing multiple design ideas for various In-game components like characters, maps, scenes, weapons, etc.To develop a single character, a designer has to factor in multiple attributes like face morphology, gender, skin tone, clothing accessories, expressions, etc leading to a long and tedious development cycle. To minimize this complexity, we aim to identify tools and techniques, which can combine the automation power of machines to generate designs based on certain guard rails defined by the designers. This approach will be a path towards machine creativity with human supervision. From a business point of view, there will be more design options for the studios to select within a short span of time leading to huge cost savings.
Our solution utilizes advantage of advance deep learning models like GANs which have been proven to work extremely well in generative tasks (generating new data instances) such as image generation, image-to-image translation, voice synthesis, text-to-image translation, etc.
In this whitepaper, we explore the effectiveness of GANs for a specific use case in the gaming industry. The objective of using GAN in this use case is to create new Mortal Kombat (MK) game characters through style transfer on new or synthesized images and conditional GANs to generate only the characters of interest.
GAN & it’s clever way of training a generative model for image generation!
GAN has two competing neural networks, namely a Generator (G) and a Discriminator (D). In the case of image generation, the goal of the generator is to generate images that are indistinguishable (or the fake images) from the training images (or the real images). The goal of the discriminator is to classify between the fake and real images. The training process aims towards making the generator fool the discriminator, hence, to get the generated fake images that are as realistic as the real ones.
Following are the highlights of the GANs solution framework to create new Mortal Kombat Game characters without using a paintbrush:
i) Showing detailed analysis and the effectiveness of GANs to generate new MK characters in terms of image quality produced (subjective evaluation) and FID distances (objective evaluation).
ii) Results of style mixing using MK characters. The style mixing performed using the trained model.
iii) Experimental evaluation using Mortal Kombat dataset (custom dataset having 25,000) images. The training time is captured to understand the computation resources required to achieve the desired performance.
Types of GAN and their role in creating Mortal Kombat realistic characters
GANs are an advanced and rapidly changing field backed by unsupervised machine learning methodology. In order to use GAN effectively to create Mortal Kombat realistic characters, it is vital to comprehend its architecture and different types in use to get near-perfect results. In this section, we will discuss the types of GAN and architectural details of StyleGans in the latter stage.
Types of GAN
1. GAN (or vanilla GAN) [Goodfellow et al. 2014]-GAN belongs to a class of methods used for learning generative models based on game theory. There are two competing neural networks, Generator (G) and Discriminator (D). The goal of GAN is to train a generator network to produce sample distribution, which mimics the distribution of the training data. The training signal for G provided by the discriminator D, that is trained to classify the real and fake images. The following is the cost function of GAN.
Cost function:
The min/max cost function aims to train the D to minimize the probability of the data generated by G (fake data) and maximize the probability of the training data (real data). Both G and D are trained in alternation by Stochastic Gradient Descent (SGD) approach.
ii) Progressive Growing GAN(ProGAN)- ProGANs (Karras et al., 2017) are capable of generating high-quality photorealistic images, starting with generating very small images of resolution 4*4, growing progressively into generating images of 8*8,…1024*1024, until the desired output size, is generated. The training procedure includes cycles of fine-tuning and fading-in, which means that there are periods of fine-tuning a model with a generator output, followed by a period of fading in new layers in both G and D.
iii) Style GANs– StyleGANs (Karras et al., 2019) are based on ProGANs with minimal changes in the architectural design to equip it to demarcate and control high-level features like pose, face shape in a human face, and low-level features like freckles, pigmentation, skin pores, and hair. The synthesis of new images controlled by the inclusion of high-level and low-level features. And it is executed by a style-based generator in styleGAN. In a style-based generator, the input to each level is modified separately. Thus, there is a better control over the features expressed at that level.
There are various changes incorporated in styleGAN generator architecture to synthesize photorealistic images. These are bilinear up-sampling, mapping network, Adaptive Instance Normalization, removal of latent point input, the addition of noise, and mixing regularization control. The intent of using StyleGan here is to separate image content and style content from the image.
(Note: For generating MK characters, we have used styleGAN architecture, and the architectural details are provided in the next section.)
iv) Conditional GANs
If we need to generate new Mortal Kombat characters based on the description provided by end-users. In vanilla GANs, we don’t have control over the types of data generated. The purpose of using conditional GANs [Mirza & Osindero, 2014] is to control the images generated by the generator based on the conditional information given to the generator. Providing the label information (face mask, eye mask, gender, hat, etc.) to the generator helps in restricting the generator to synthesize the kind of images the end-user wants, i.e. “content creation based on the description”.
The cost function of conditional GAN given below:
Cost function:
This conditional information is supplied “prior” to the generator. In other words, we are giving an arbitrary condition ‘y’ to the GAN network, which can restrict G in generating the output and the discriminator in receiving the input. The following figure depicts the cGAN inputs and an output.
Fig 1. A- Random generation of MK characters using Random Noise vector as input, B-Controlled generation of MK characters using Random Noise vector and labeled data as input (Condition-Create MK characters with features as Male, Beard, Moustache)
Architectural details of StyleGANs
Following are the architectural changes in styleGANs generator:
The discriminator D of a styleGAN is similar to the baseline progressive GAN.
The generator G of a styleGAN uses baseline proGAN architecture, and the size of generated images starts from 4*4 resolution to 1024*1024 resolution on the incremental addition of layers.
Bi-linear up/down-sampling is used in both discriminator and generator.
Introduction of mapping network, which is used to transform the input latent vector space into an intermediate vector space (w). This process is executed to disentangle the Style and Content features. The mapping function is implemented using 8-layers Multi-layer Perceptron (8 FC in Fig 2).
The output of the mapping network is passed through a learned Affine transformation (A), that transforms intermediate latent space w to styles y=(ys, yb) that controls the Adaptive Instance Normalization module of the synthesis network. This style vector gives control to the style of the generated image.
The input to the AdaIN is y = (ys, yb) which is generated by applying Affine transformation to the output of the mapping network. The AdaIN operation defined as the following equation:
Each feature map xi is normalized separately and scaled using the scalar ys component and biased using the scalar yb component of y. The synthesis network contains 18 convolutional layers 2 for each of the resolutions for 4×4 – 1024×1024 resolution images. So, a total of 18 layers are present.
7. The input is a constant matrix of 4*4*512 dimension. Rather than taking a point from the latent space as input, there are two sources of randomness induced in generating the images. These results were extracted from the mapping network and the noise layers. The noise vector introduces stochastic variations in the generated image.
a) ProGAN generator
b) StyleGAN generator
Fig 2. a-ProGAN generator, b-StyleGAN generator [Karras et al., 2019]
Following is the table depicting different configuration setups on which style GANs can be trained.
Table 1: Configuration for StyleGANs
Business objectives and possible solutions
Experimental setup
Dataset
Fig 3. Sample images from the training MK dataset
Experimental design
We have conducted a set of experiments to examine the performance of StyleGANs in terms of FID, quality of output produced, training time vs performance on FID. In addition, we also checked the results imposing pre-trained latent vectors on new faces of data and Mortal Kombat characters data. We have implemented GANs and performed its feasibility analysis to overcome the following issues:
i) How effective StyleGANs are in producing MK characters with lesser data, low- resolution images, lighter architecture, and less training?
ii) How computationally extensive GANs are? Are GANs expensive to train? How to estimate the time and computational resources required to generate the desired output?
iii) How well the pre-trained vectors used for style-mixing to the original images?
Evaluation metrics
FID [Heusel et al. 2017]-Frechet Inception Distance score (FID) is a metric for image generation quality that calculates the distance between feature vectors calculated for real and generated images. FID is used to understand the quality of the image generated, the lower the FID score, higher the quality of image generated. The perfect FID score is zero. Experimental platform- All experiments are performed on the AWS platform, and the following is its configuration.
Experiments results
In order to provide a precise view of image generation of MK characters, we have conducted various experiments; and we were able to extract different results for each experiment.
Training- The training images retrieved at a different number of kims are depicted in the following figures. All experiments were conducted at configuration “d”. Refer to Table 1. Fig 4. MK training results 1-Kims 2364, 2-Kims 5306 , 3-Kims 6126, 4-,Kims 8006 5-Kims 9766, 6- 10000 Kims
Training time– The following table depicts the time taken, and FID results for the model trained till 10000 kims. The best-trained models have an FID of 4.88 on the FFHQ dataset (Karras et al., 2019) on configuration d (refer table 1). We got the final FID score of 53.39 on the MK dataset.
Table 3. Time taken for training & FID score
iii) Real-time image generation – Following are some of the fake images generated using different trained models using random seed values.
Using the Style Mixing method to generate new variations in the character appearance where we could use two or more reference images to generate new results. In this context, we have used two characters to create a new character.
Fig 6. Style Mixing results on
MK characters
v) Progressively growing GAN- results on MK
When progressively growing GAN is applied to MK characters, with its nature of incrementally adding the layers to the network, the network learns to generate smaller and low-resolution images, followed by more complex images. Thus, stabilizing the training and overcoming mode collapse of the generators.
Fig 7. Progressive GANs output
Conclusion
In this whitepaper, we have discussed the methodology of applying the feasibility analysis using styleGANs for Mortal Kombat characters. We have provided a detailed report on the GANs types and evolution with respect to image generation approaches to solve different use cases using GANs. Which also includes style representation and conditional generation. The latter part of our attempt provides the results of GANs training time, FID scores, real-time image generation output, style mixing, and ProGANs results. After completing ~15 days of training (with GPU cost $1.14 per hour), the FID score achieved is 53.39. The quality of images is also improved, which can be further enhanced by conducting flawless training sessions. Recent advancements
in GANs illustrate that a better GAN performance is achievable even with fewer data. Adaptive Discriminator Augmentation [Karras et al., 2020] and Differentiable Augmentation [Zhao et al., 2020] are a few of the recent approaches which have been proposed to train GANs effectively even with less amount of data, which is currently being researched in our CoE team.
About Affine
Affine is a Data Science & AI Service Provider, offering capabilities across the analytical value chain from data engineering to analytical modeling and business intelligence to solve strategic & day-to-day business challenges of organizations worldwide. Affine is a strategic analytics partner to medium and large-sized organizations (majorly Fortune 500 & Global 1000) around the globe that creates cutting-edge creative solutions for their business challenges.
References
[1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).
[2] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems (pp. 6626-6637).
[3] Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196.
[4] Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4401-4410).
[5] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data.arXiv preprint arXiv:2006.06676.
[6] Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets.arXiv preprint arXiv:1411.1784.
[7] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable Augmentation for Data-Efficient GAN Training. arXiv preprint arXiv:2006.10738.
Uber has revolutionized the way of commute since its launch. Traveling short distances has never been hassle free. Earlier people used to use their personal vehicles to cover small distances. Other alternative was to use public transport which is time-consuming and inconvenient. Uber, on the other hand, provides flexibility to non-frequent traveler and ones who love commuting over shorter distances, as they do not have to spend on purchasing a vehicle and at the same time can move around very conveniently. The same might hold true for the future of gaming! What would you feel if technology giants like Google and Amazon-owned the expensive hardware to process games with the best possible CPU and GPUs allowing you to simply stream the games? This could potentially eliminate the need of purchasing an expensive console and pay in a proportion of usage! This could be a game changer especially for someone who has not been able to commit to a INR 30,000/- console to play a single game. Can the entry of Google and Amazon in the gaming industry make this possible? At the Game Developers Conference (GDC) 2019, Google unveiled its cloud-streaming service called STADIA. Just like how humans have built stadiums for sports over hundreds of years, Google believes it’s building a virtual stadium: Stadia, to foster 1000s of player to play or spectate games simultaneously interacting with each other. Free to play games like Fortnite will standout on Stadia if Google can increase the number of players participating in an instance from 100 to say 1000s. Would Stadia really live up to its hype is a tricky question that only time may answer.
2. How does it work?
Google will make use of its massive data centers across the globe that will act as computational power for this service. Massive servers will make use of its advanced CPUs, GPUs, RAM and ROM to render games and stream to the users the enhanced audio/visual outputs. The players’ input shall be uploaded via keyboard or custom Stadia controller directly to the server. Let’s look at how Stadia stands against conventional console-based gaming.
3. Comes with advantages over console-based gaming
3.1. No Hardware (other than a remote): The bare minimum piece of hardware required is a device that can run chrome like a laptop, PC, mobile, tablet or even smart TV.
3.2. No Upgrade costs as they are taken care of, by the shared infrastructure hosted by Google. In the recent past, we had games that were below 10 GB in size while the recent RDR2 was above 100 GB with its patches. One can imagine how the need to upgrade hardware is the biggest driver for upgrading to next-gen consoles.
3.3. No RAM/ROM or Loading time limitations: Apart from these, YouTube integration will enable users to live broadcast their gameplay and will allow others to join as well in case of multiplayer games. In addition, the google assistant present on stadia controller will provide immediate help in case one is stuck at some point
of time to clear the stage. The benefits of this concept are really promising. But will the drawbacks offset these promises? Let’s go through each of them.
4. Need to overcome challenges to expand at scale
The drawbacks can potentially be addressed over time, but for now, scaling this remain the biggest hindrance. There are various challenges that Google (and users) will face such as Latency, Pricing, Markets and Game Library. There are other pointers as well, but these are going to be the biggest ones.
4.1. Latency effect
The video footage must get to you and the controller inputs must get from you to the server. Hence it is obvious that there is going to be an extra latency. Latency will depend upon three elements:
– Amount of time to encode and decode the video feed: Google has tons of experience in the field of video feed under the likes of YouTube
– The quality of internet infrastructure at the end user: This worrisome problem will hinder the smooth conduct of this process. The internet speed will be good in tier 1 cities, and not necessarily in the rural areas. You will also need a data connection without any cap. As per google, a minimum speed of 25Mbps will be required to bring Stadia into function. This means 11.25 GB of data will be transferred per hour. That’s about 90 hours of game streaming before the bandwidth is exhausted, considering that the user has a data cap of 1 TB. In other words, 3 hours of gaming per day in a month of 30 days. This is under the assumption that there is only one user and is utilized only for gaming purpose.
4.2. Dilemma for developers
Above was the issue that the end user will face. Let’s look at the situation from the game developer’s perspective. With the advent of a new platform, the developers will have yet another platform to port and test games. The developers will have to do more research which will increase the cost of production. At the same time, more time will be required to release game. This will be a big challenge for franchises that launch games every year. Google has partnered with Ubisoft and has promised to feature Ubisoft games at launch. The time will tell how many more developers will be willing to go a step ahead to support this concept. If not, then this could potentially mean that a lot of games will not be available ever. Now from a consumer’s perspective, it will be hard to justify their purchase as they won’t be able to play all the games available in the market.
4.3. Optimal pricing
Another challenge will be pricing. There is no information regarding the pricing of the overall model. Is this going to be a subscription service? Do we have to buy games? How the revenue is going to be shared with developers? Will the pricing be the same for hardcore gamers and casual gamers? Consider Activision (developer of games like Call of Duty) for example. Historical analysis tells us that slightly more than one-fourth of the purchasers do not even play the game for few hours. On the other hand, there are purchasers who play it day in and day out. The cost that each user has to pay for the game is $60. This amount goes to Activision and the platform on which it is sold. In case, Activision decides to release the game on Stadia, all the casual purchasers who would have bought the game to test out the hype, would now just stream it on Stadia at a much lower cost. Will Activision take that chance and release the game on Stadia? In case, the pricing is different for the types of users, how will the revenue be shared with the developers? Let’s assume that this will be a subscription model and users will be charged $30 per month, which comes out to be $360 per year. Now for a casual gamer, this will be very high as he can buy a console for $300 and play for years. All these questions will have to be answered before the launch. Running a cloud gaming service is
expensive. If the whole selling point is making gaming accessible to more and more people, then a high price point is not going to help the cause.
4.4. Available markets
At the GDC event, the team said that the service will be available in the US, Canada, UK, and Europe at launch. These regions have a high penetration of console-based gamers and Google will have to make a lot of efforts to make these people switch. The penetration of PlayStation and Microsoft Xbox is in single digits in India or China. With Stadia not available in Asia, Google is missing a lot of developing countries like India and China where people are not inclined towards consoles and hence hampering its user coverage. Given the high cost of consoles in developing countries like India, Stadia can become the go-to gaming platform.
4.5. Array of games available
Games library will be another hurdle in the race. We have no information regarding the list of games available during launch. Third party support isn’t enough for a gaming platform to survive. You need a list of exclusive games to bring people aboard. Google even unveiled its own Stadia Games and Entertainment studio to create Stadia-exclusive titles, but it didn’t mention any details on what games it will be building. In addition, it is highly unlikely that Console exclusives (1P titles) like Spider-Man or Halo will be available for Stadia. 1P games play a significant role in the console sales and Sony and Microsoft will never let this happen until they stick to console-based gaming. So, Google will have come up with its own exclusive titles so be dominant in the market. Making exclusive games takes a lot of research and time. It took Sony a good 5-6 years to develop one of its best-selling game “God of War”. If Google has not already started on its exclusive games, then it would be a mountain to climb for them.
4.6. What about other browsers?
Stadia will only be available through Chrome, Chromecast, and on Android devices initially. There was no mention of iOS support through a dedicated app or Apple’s Safari mobile browser. Will Apple be comfortable to let its user base shift completely to Chrome from Safari? Will Apple charge Google additional money for the subscription that Google gets on Apple’s devices? All these questions will be answered over time.
4.7. What if…?
Last but not the least, in case Google decides to drop the idea of Stadia in the later years of its launch like it has done in the past with Google lens or google plus, then gamers will lose all their progress and games despite their subscription fees. Apart from the above drawbacks, Google is not the only company to step in this field. It already has some serious competition from existing players in the game streaming sector.
5. Any competition that Google might face?
Sony already streams games to its consoles and PCs via its PlayStation Now service. Microsoft is also planning its own cloud game streaming service and can leverage its Azure data centers. Also, both Sony and Microsoft don’t require developers to port their games for their cloud streaming service. Apart from these two players, Nvidia has been quite successful in this domain allowing users to stream games from its library. This means Google has some strong competition and looks like the cloud gaming war is just getting started.
6. Conclusion
What is the incremental change you get from one version of a device to another? It is the absolute bare minimum they can give to make people switch. Let’s take an example of PS4 slim and PS4 pro. The only difference is that Pro supports 4K while Slim doesn’t and we have seen 30% people switching from Slim to Pro. The entrance of Google into the gaming industry will make PlayStation better, it will make Xbox better, it will make internet infrastructure better. The success or failure of Google stadia will cost nothing to consumer and at the same time, it will be net positive to gaming industry as well.
Thanks for reading this blog, For anyfeedback/suggestions/comments, please drop a mail to marketing@affine.ai
Affine recently completed 6 years, I have been a part of it for about 3 of those years. As an analytics firm, the most common business problem that we have come across is that of forecasting consumer demand. This is particularly true for Retail and CPG clients.
Over the last few years have dealt with simple forecasting problems for which we can use very simple time-series forecasting techniques like ARIMA and ARIMAX or even linear regression these are forecasts which are more at an organization or for specific business divisions. But over the years we have seen a distinct shift in focus of all our clients to get forecasts at a more granular level, sometimes for even specific items. These forecasts are difficult to attain using simple techniques. This is where more sophisticated techniques come into play. These techniques are the more complex machine learning techniques which include RF, XG Boost etc.
We cater to various industry domains and verticals and to explain how the clients’ requirements have changed over the years I can think of two very distinct examples from two specialty domains, video gaming industry and sportswear manufacturer and retailer. Below I will try to explain how the business requirement for a forecast was different for both these clients.
Video Game Publisher
Over the last few years, the popularity of one franchise belonging to the publisher has gone down due to various factors. The stakeholders wanted to understand the demand pattern for the franchise going forward and they wanted monthly predictions for the franchise’s sales for the next 1 fiscal year. This franchise contributed to almost 60% – 70% of the organization’s revenue and we were required to predict the sales for only this franchise. Also since this was a month level forecast/prediction we had which are primarily black box techniques being appropriate for this requirement. enough data points to use either a time series analysis or even a regression analysis to predict sales. We tried both and finalized on a regression-based analysis so that we could also identify the drivers for sales and their impact which was important for the stakeholders.
Sportswear Manufacturer and Retailer
In the case of the sportswear manufacturer and retailer client, they wanted weekly forecasts for all the styles available in their portfolio. Hence the client required predictions for all the items available for all the weeks in a fiscal year.
There are a lot of items here which are newly launched items having very few data points at a week level. Here, the traditional time-series methodology will fail because of lack of data points, also not all the styles will showcase similar trend and seasonality. Along with this, there will also be styles which have minimal sales and prediction for these styles is a major challenge for this client. We had to develop an ensemble of models where we divided all the styles into few buckets of
High volume – high duration
High volume – low duration
Low volume – high duration
Low volume – low duration and completely new launches.
For the styles that have high volume and high duration, we can still use a time-series or a regression technique but for all the others these traditional methods will have limitations. Hence, we needed to apply ML techniques for these styles.
For the styles with low duration, we used Random forest and XGB methods to arrive at the predictions. Also for these styles what was more important was to get a proper demand prediction rather than identify the drivers of sales and their impact, hence ML techniques.
Conclusion
As an analytics practitioner, we recognize that there is no one-size-fits-all approach to data analysis. To identify the best approach, one needs to have a deep knowledge and practical experience in various approaches. This was established in our recent experiences. While the video game publisher’s requirement was primarily for an entire franchise for which we could use simple time series and regression techniques, the sportswear retailer’s requirement was much more granular. In another experience with the sportswear retailer, item level predictions were the prime requirement and over time we as an analytics firm have seen this change in requirement from an overall demand prediction too much more granular predictions across the board. Also, most of our clients tend to make informed decisions about how much inventory to produce and stock for each item and a granular prediction at an item level aids that.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
