What type of method are you using to insert data into the destination DE? (Overwrite, Add, Update)
If Overwrite - you can try using a (Select DISTINCT * FROM [TABLE]) to make sure the result of your query is truly distinct and you don't have duplicate keyed records somewhere.
If Add - the data could already exist in the DE - try doing an Add/Update or just an Update instead or you can add an Where NOT EXISTS clause in your query...
SELECT
a.ID as ‘ID’,
a.System as ‘System’,
a.Updated as ‘Updated’,
a.LocalId as ‘LocalId’,
‘TRUE’ as ‘IsDeleted’
FROM
[Contact Source] a
Inner JOIN
DeletedInvalid_Staging b
on a.ID=b.ID
WHERE NOT EXISTS (Select 1 FROM [Your Destination DE] t1 WHERE a.ID = t1.ID AND a.System = t1.System)
If Update - keep in mind, the result of the query you are inserting isn't inserting record by record - overwriting a previous record it just updated. The results have to be distinct.